Pointing Devices
Two types of pointing devices are generally used for interactive applications. With indirect pointing devices, e.g., mice, touch pads and joysticks, the movement of the pointer on the display surface does not correspond to the movement of the pointing device. For example, a mouse moves horizontally, while the pointer or cursor on the display screen moves vertically. This causes a perceptual disconnect between the user, the pointing device and the application. With direct pointing devices, e.g., laser pointers and styli, the movement of the pointer, i.e., a red dot, corresponds exactly to the movement of the pointing device in the hand. This greatly enhances the interaction. However, direct pointing devices suffer from jitter and lack of precision.
Projecting Devices
Projectors have traditionally been used as static, output-only devices for presenting content in a non-interactive manner to a relatively passive audience. However, recent advances in projection technology have led to significant decreases in cost, size, and power requirements. This is a trend that is likely to continue.
Users can now consider more interactive and mobile uses for projectors. Consider the possibilities that arise when, for example, the projector is small and light enough to be handheld. In this case, any convenient surface can become a computationally enabled display. Unlike conventional small display screens used with portable computing devices such as mobile telephones and personal digital assistants (PDAs), a projected image is large enough to enable viewing by more than one person.
Raskar et al. describe applications for handheld projectors and a variety of interesting ways in which the projectors can be used in an interactive manner, Raskar et al., “iLamps: Geometrically Aware and Self-Configuring Projectors”, ACM Transactions on Graphics (TOG), ISSN: 0730-0301, vol. 22, issue 3, pp. 809-818, July 2003, also in: Mitsubishi Electric Research Laboratory (MERL) Technical Report, TR2003-023; U.S. Pat. No. 6,764,185, issued to Beardsley et al. on Jul. 20, 2004, “Projector as an input and output device”; and Raskar et al., “RFIG lamps: interacting with a self-describing world via photo-sensing wireless tags and projectors”, ACM Transactions on Graphics, 23(3), pp. 406-415, 2003.
Of particular interest is their solution to the problems of keystone correction and rotation. By using a camera in conjunction with the handheld projector, they determine a pose of the projector relative to the display surface and use the pose to correct for distortion and rotation, and to factor out projector motion. This enables a dynamically updated output image to be stably displayed at a fixed location and orientation.
Pointing Projectors
By displaying a pointer or cursor at the center of the output image, movement of a projector can directly control the pointer's movement across a stabilized region inscribed within the projector's output image. With direct pointer control and a button on the projector, all standard interactions as in a ‘WIMP’ interface are possible. This results in an interactive handheld projector that is concurrently an output device and an input device.
Furthermore, because the pointer directly follows hand movement, the interaction is akin to direct pointing, which is arguably more satisfying than other indirect pointing, such as using a mouse, a touch pad, a roller ball or a joystick.
Although appealing in its simplicity and directness, this style of interactive handheld projection has limitations in practice due to the lack of precision of pointer movement, as well as low resolution of the stable image within the output image.
A number of techniques are known for using laser pointers as input devices for large screen interaction, C. Kirstein and H. Muller, “Interaction with a projection screen using a camera tracked laser pointer”, Multimedia Modeling Conference., p. 191-192, 1998; S. Matveyev and M. Göbel, “The Optical Tweezers: multiple-point interaction technique”, Virtual Reality Software and Technology. p. 184-188, 2003; S. Matveyev, M. Göbel and P. Frolov, “Laser pointer interaction with hand tremor elimination”, HCI International, p. 376-740, 2003; B. Myers, R. Bhatnagar, J. Nichols, C. H. Peck, D. Kong, R. Miller and C. Long, “Interacting at a distance: measuring the performance of laser pointers and other devices”, ACM CHI Conference on Human Factors in Computing Systems, p. 33-40, 2002; J.-Y. Oh and W. Stuerzlinger, “Laser pointers as collaborative pointing devices”, Graphics Interface, p. 141-149, 2002; D. R. Olsen and T. Nielsen “Laser pointer interaction”, ACM CHI Conference on Human Factors in Computing Systems, p. 17-22, 2001; and C. Peck, “Useful parameters for the design of laser pointer interaction techniques”, Extended Abstracts of the ACM CHI Conference on Human Factors in Computing Systems, p. 461-462, 2001.
Jitter
Laser-pointer interaction is hindered by the same handheld jitter problem faced by interactive handheld projection. Myers et al. compared laser pointers to other devices in pointing tasks and found laser pointers to perform the worst with, at best, a four pixel selection accuracy even after predictive filtering. Oh et al. designed a computer-controlled laser pointer with Kalman filtering. However, they still had error rates of around 40% when selecting relatively large 40-pixel diameter targets. Matveyev et al. describe a more elaborate model for reducing the effects of jitter but do not provide user data measuring the performance of their approach. Peck describes significant handheld jitter effects for laser-pointer tracking deviations. The Olsen et al. laser-pointer interaction technique is optimized to avoid handheld jitter as much as possible.
The prior art seems to indicate that the effects of hand-jitter are not easily solved by data-filtering techniques.
Delay
More importantly, techniques that use data-filtering, even if they do work, introduce a significant delay. Delay hinders user interaction. That is, the physical cues are not concurrent with the visual cues. This is detrimental to performance, see C. Ware and R. Balakrishnan, “Reaching for objects in VR displays: Lag and frame rate”, ACM Transactions on Computer-Human Interaction, 1(4), pp. 331-356, 1994.
Standard delay-inducing, pointing-enhancing techniques, such as using non-linear transfer functions that change pointer-movement characteristics based on the velocity of the user input, work reasonably well for indirect input devices, such as a mouse, a touchpad, or a joystick.
However, those techniques are not easily applicable to direct pointing devices, such as interactive handheld projectors, styli and laser pointers, where there is a direct one-to-one correspondence between device movement and pointer control. Using non-linear transfer functions in those scenarios introduce a dynamically changing offset between device and pointer positions, resulting in a loss of the very directness that makes such pointing techniques so attractive in the first place. Furthermore, if the offset becomes too large, the pointer can end up outside the tracking envelope.
Pointer prediction techniques have been successfully used to stabilize indirect pointing, J. Baldwin, A. Basu and H. Zhang, “Predictive windows for delay compensation in telepresence applications”, IEEE International Conference on Robotics & Automation, pp. 2884-2889, 1998 and A. Murata, “Improvement of pointing time by predicting targets with a PC mouse”, International Journal of Human Computer Interaction, 10(1), pp. 23-32, 1998. However, those techniques are difficult to apply to direct absolute pointing due to the resulting disconnect that arises between actual and predicted pointer positions when delays occur.
Precision
Zooming and magnification techniques have been used to explore areas of interest in an image, see G. Furnas, “Generalized fisheye views”, ACM CHI Conference on Human Factors in Computing Systems. pp. 16-23, 1986 and M. Sarkar and M. Brown, “Graphical fisheye views of graphs”, ACM CHI Conference on Human Factors in Computing Systems, pp. 83-91, 1992.
A variety of focus-in-context techniques are also known, M. S. T. Carpendale and C. A. Montagnese, “A framework for unifying presentation space”, ACM UIST Symposium on User Interface Software and Technology, pp. 61-70, 2001; S. Carpendale, “A framework for elastic presentation space”, Department of Computer Science, Simon Fraser University, 1999; J. Mackinlay, G. Robertson and S. Card, “The perspective wall: Detail and context smoothly integrated”, ACM CHI Conference on Human Factors in Computing Systems, pp. 56-63, 1991 and C. Ware and M. Lewis, “The DragMag image magnifier”, Extended Abstracts of the ACM CHI Conference on Human Factors in Computing Systems, pp. 407-408, 1995.
Another technique uses tracking menus, G. Fitzmaurice, A. Khan, R. Pieke, B. Buxton and G. Kurtenbach, “Tracking menus”, ACM UIST Symposium on User Interface Software and Technology, pp. 71-79, 2003. There a pointer moves freely within a predefined region, but drags the region along in the direction of pointer movement when the pointer moves beyond the edge of the region.
It is desired to solve the jitter and precision problems for handheld pointers.